A Study on High-quality View Synthesis for Free-viewpoint Television

Free-viewpoint Television (FTV) is becoming an exciting media that enable us to view a 3D scene by freely changing our viewpoint. Considered as ultimate 3DTV, FTV can provide arbitrary views using captured real views. Essentially, it is a re-sampling problem in the ray-space or light field. The key to address this re-sampling problem is synthesizing virtual views by captured reference views along with geometry information. In this thesis, we study how virtual views can be reliably generated from multiple captured videos for virtually viewpoints exploring in our world. One key challenge is that the required geometry information may contain geometry errors. As the most common geometry information, the depth map is obtained by stereo matching or other geometry extraction techniques to provide the 3D location for each pixel. The erroneous depth information will cause incorrect and uncomfortable boundary artifacts in view synthesis results. This thesis introduces a novel analysis for different artifacts with respect to different depth errors. The proposed artifact category is the first to model and categorize various view synthesis artifacts and depth errors in FTV or 3DTV. This contribution establishes the exact relationship between a particular artifact case and a specified geometry error mode with respect to the boundary. Using this category, we find a complementary property for artifacts with symmetric reference views, which can be employed to design the artifacts compensation mechanism. This means that the artifacts introduced from one reference can be compensated by the other reference. Applying artifacts compensation is nontrivial since it requires knowing which pixels are erroneous or unreliable for view synthesis. Reliability information addresses the problem of spatial ambiguity in view interpolation to suppress boundary artifacts. The reliability computation is determined by potential synthesis errors and the erroneous pixels which cause synthesis errors or artifacts will be labeled as unreliable. For reliability reasoning, the potential synthesis error is a particular concern, as the synthesized view is unknown. We introduce the reference crosscheck to infer the potential synthesis error and define the reliability adaptively for each pixel in reference views. Reliability reasoning is conducted both deterministically and probabilistically. The deterministic reliability is in the binary form while the probabilistic reliability is continuous. Furthermore, the binary reliability is the simplified version of its continuous counterpart. The continuous reliability can further suppress noises in synthesized views. Considering view synthesis as an image reconstruction problem and following Bayesian inference, two fundamental reliability-based view synthesis solutions are provided. Without regularization, the first solution is exactly the optimum of a Maximum likelihood (ML) problem. It has the close form and can be solved by linear equation in real-time. The other regularization based solution is solved in the Maximum a Posterior (MAP) framework by graph cuts. A hybrid solution is also available by combining the previous two solutions. The regularization based method can generate virtual views with least artifacts and best quality. Meanwhile, the solution without regularization is computationally efficient and can be utilized to replace the conventional virtual view interpolation. It is also able to tolerate heavy video compression thus can save the required bitrates. The probabilistic reliability-based view synthesis method can save 10% bitrates at the decoder side of the FTV/3DTV transmission system. A second challenge is to synthesize close-up virtual views. Unlike common intermediate virtual views where the resolution is generally similar to captured reference views, close-up virtual views are in higher resolutions than reference views. The resolution expansion causes new holes and artifacts. The holes are spread all over the synthesized views and require expansive inpainting to fill. Furthermore, the newly arising artifacts exist in boundary areas since foregrounds can not completely occlude backgrounds due to holes. To properly handle the new problem, we firstly analyze the occlusion problem in closeup virtual view synthesis. Then, occlusion artifacts are explicitly labeled by a reliability filter. Finally, the novel reliability based 3D warping technique is developed to eliminate noticeable artifacts and holes. We have also developed a one-to-many 3D warping approach for close-up view synthesis. In contrast to previous work on one-to-one view synthesis, the one-to-many view synthesis method estimates perpixel resolution magnification factor and expands the warped pixels adaptively according to the resolution expansion ratio. It is able to automatically handle previous occlusion failure and holes in real time. We show that the proposed view synthesis can significantly reduce visible artifacts and improve the quality of the synthesized close-up view. Together these techniques enable the reliable view synthesis to provide plausible virtual views for the immersive virtual viewpoint display in FTV. We can provide virtual views with less artifacts, higher PSNR, lower bitrates and wider navigation range.